Matrix kappa: a Proposal for a Card Sort Statistic for is Survey Instrument Development

The card sort is a key scale development tool that is frequently used in IS survey instrument development. Cohen's Kappa is a recommended measure of inter-rater agreement in this process, however one of its underlying statistical assumptions is violated when it is used in open card sorts. To address this issue, Matrix Kappa is proposed as a complement to other card sort analysis techniques, reframing constructs in terms of item relationships and representing inter-rater agreement in terms of matrices. Matrix Kappa has the benefit of meeting Cohen’s Kappa assumptions for open card sorts and can be used to differentiate both open and closed card sort results that Cohen’s Kappa cannot.